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Water entered the Extravehicular Mobility Unit (EMU) helmet during extravehicular 
activity (EVA) #23 aboard the International Space Station on July 16, 2013, resulting in the 
termination of the EVA approximately 1 hour after it began. It was estimated that 1.5 liters 
of water had migrated up the ventilation loop into the helmet, adversely impacting the 
astronaut’s hearing, vision, and verbal communication. Subsequent on-board testing and 
ground-based test, tear-down, and evaluation of the affected EMU hardware components 
determined that the proximate cause of the mishap was blockage of all water separator 
drum holes with a mixture of silica and silicates. The blockages caused a failure of the water 
separator degassing function, which resulted in EMU cooling water spilling into the 
ventilation loop, migrating around the circulating fan, and ultimately pushing into the 
helmet. The root cause of the failure was determined to be ground -processing shortcomings 
of the Airlock Cooling Loop Recovery (ALCLR) Ion Filter Beds, which led to various levels 
of contaminants being introduced into the filters before they left the ground. Those 
contaminants were thereafter introduced into the EMU hardware on-orbit during ALCLR 
scrubbing operations. This paper summarizes the failure analysis results along with 
identified process, hardware, and operational corrective actions that were implemented as a 
result of findings from this investigation. 
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Nomenclature 


ACTEX 

= 

Activated Carbon/Ion Exchange 

ALCLR 

= 

Airlock Cooling Loop Recovery 

CAP 

= 

Corrective Action Plan 

D1 

= 

deionized 

ECLSS 

= 

Environmental Control Life Support System 

EMU 

= 

Extravehicular Mobility Unit 

EVA 


extravehicular activity 

Ee 


iron 

EPS 


fan/pump/separator 

ISS 


International Space Station 

JSC 


Johnson Space Center 

LCVG 

= 

Liquid Cooling and Ventilation Garment 

mL 

= 

milliliter 

Ni 

= 

nickel 

pH 

= 

hydrogen ion concentration 

ppm 

= 

parts per million 

SEMU 

= 

Short Extravehicular Mobility Unit 

Si 

= 

silicon 

TOC 

= 

Total Organic Carbon 

Zn 

= 

zinc 


I. Introduction 

W ater entered Astronaut Luca Parmitano’s helmet during U.S. extravehicular activity (EVA) 23 on July 16, 
2013, on-board the International Space Station (ISS), resulting in early termination of the EVA at a phase 
elapsed time of 1:05. Water entered the helmet at the ventilation loop inlet “T-2” port, which is located behind the 
crew member’s head at the base of the helmet. The specific location where the water was entering and the source of 
the water was unknown during the EVA. The water began to accumulate on the back of Luca’s head and he soon 
reported that the water was increasing, Mission Control Center-Houston directed the crew to terminate the EVA. 
The water migrated to Luca’s face during translation back to the airlock, thereby covering his eyes, ears, and nose. 
Luca’s vision and communications were degraded as a result of the water shift, requiring Luca to use his safety 
tether as a guide to navigate back to the Airlock. The crew successfully returned to the Airlock and performed an 
expedited repress and suit doffing. Luca communicated with EVl, Chris Cassidy, during repress using only hand 
squeezes. Eollowing EVA 23, the crew reported approximately 1.5 liters of water in Luca’s helmet — a fact that was 
later confirmed when the Extravehicular Mobility Unit (EMU) feed-water recharge was performed as part of the 
standard post EVA suit servicing.^ 

This paper summarizes the investigative findings that lead to the proximate and root cause, provides associated 
corrective actions, and outlines recommended forward work related to the EVA 23 water in the helmet mishap. The 
mishap investigation, including the development of proximate and root cause for the failure and implementation of 
corrective actions, culminated in returning the ISS Program to nominal EVA capability on August 1 1, 2013. 

II. On-orbit Hardware Evaluation 

On-orbit troubleshooting of the failed EMU began the day following the EVA and included eliminating the 
Liquid Cooling and Ventilation Garment (LCVG), water tank structure, and hard upper torso tubing as leak sources. 
The failure was easily repeated on-orbit in the intravehicular activity environment by pressurizing the fully-suited 
EMU and “priming” the Transport (e.g., Cooling) Water Loop to initiate cooling loop water flow through the water 
separator. During this troubleshooting, water entered the unmanned helmet within about 3 minutes of priming. A 
fault tree was developed and more complex troubleshooting, including removal and replacement of various Life 
Support System components, was conducted through late October 2013. This troubleshooting eventually led to the 
removal and replacement of EMU 301 1 ’s fan/pump/separator (EPS) (Item 123 S/N 006) on October 24, 2013, which 
resolved the water crossover anomaly. The EPS became a focal point of the investigation going forward and was 
returned to the ground for post- flight analysis. The S/N 006 EPS was removed from the Short Extravehicular 
Mobility Unit (SEMU) 3011 on orbit and returned to ground for test, teardown, and evaluation on 11/11/2013. This 
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approach was simple: move from least-invasive to most-invasive actions during the teardown of FPS 006 to preserve 
as much evidence as possible for evaluation. ^ 

The EMU fan, pump, and water separator are coupled together through a common shaft and magnetic coupling 
to form what is called the FPS (Figure 1). The fan circulates oxygen over the crew member’s head and throughout 
the suit, and pulls waste gases (e.g., carbon dioxide) and oxygen through a carbon-dioxide-removal cartridge. The 
pump, which is isolated from the ventilation loop via a magnetic coupling, circulates coolant loop water through the 
EMU’s sublimator heat exchanger and throughout the LCVG. Humidity from the crew member’s' exhalation and 
sweat is also condensed in the sublimator and sent to the water separator where the water is removed and returned to 
the water loops. In addition to this condensate, water from the EMU coolant loop is continuously flowing through 
the water separator to facilitate constant coolant loop circulation for degassing purposes.^ 



1-1 23B Water 
Separator 


Figure 1. Cutaway view of the EMU FPS. 


III. Ground Hardware Evaluation 

A receiving inspection was performed when the S/N 006 FPS assembly arrived at the United Technologies 
Aerospace Systems (UTAS) facility and the Item- 123 FPS assembly was separated from the Item- 127/1 28 Pump 
Inlet Filter. Both were repackaged and shipped to Goddard Space Flight Center for X-ray CT scans. The scans were 
primarily looking for mechanical assembly or component failure areas or gross contamination that may have caused 
the failure of the FPS to function properly. The result of the nondestructive evaluation showed no mechanical issues 
such as cracks or loss of a weld plug in the pitot that would result in the failure mechanism seen on orbit. However, 
both the X-ray CT scan and N-ray scan showed the presence of contamination within the drum of the separator. 
Figure 2 shows the contamination indications in the drum from the CT scans. The contamination appeared to be in 
and around the eight drum holes that feed the drum trough.^ 
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Figure 3: X-ray CT scan of FPS S/N 006 separator drum. 


After the CT scans were complete, the hardware was sent back to UTAS for disassembly. All parts were 
photographed and dimensionally inspected during disassembly. Once the FPS pitot and drum were removed, they 
were packaged and shipped to McClellan Nuclear Research Center for N-ray scans. The N-ray scan was to 
determine whether internal passageways within the pitot and drum were blocked or contained contamination. 
Testing of the water separator was added to the plan, based on findings from the inspections and nondestructive 
evaluations. 

Upon return of the FPS, visual inspections identified contamination in all eight water separator drum inlet feed 
holes (Figure 3) and along the walls leading to the feed holes. Analysis showed the material to be primarily loosely 
bound silica, with zinc acetate, aluminum silicate, and aluminum oxide present as well. These constituents are all 
native to the EMU/Airlock system, but the quantity and location were out of family. The majority of the material 
was silica based. ^ 

In operation of the EMU Transport Water Loop, small particles are filtered with a 20 micrometer filter prior to 
entering the water separator, which indicates the contaminant material either entered as very small particles that 
agglomerated in the separator or entered in solution followed by localized precipitation. Testing of EMU 301 Us 
returned FPS (S/N 006) both with and without the drum-hole contamination confirmed that the blocked drum holes 
were the cause of the water in the helmet mishap. 
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Figure 3. Blocked water separator drum holes. 


After the silica-laden contamination was found in the FPS, a causal tree was developed to determine the source 
of the silica and the potential mechanisms for facilitating the precipitation and agglomeration in the water separator. ^ 
All hardware and water sources that interface with the EMU were investigated. This included EMU 301 1’s interface 
with the Space Shuttle coolant loop prior to launch, Space Shuttle water from the Payload Water Reservoirs, which 
are used to fill the EMUs on the ISS, the Airlock Cooling Loop Recovery (ALCLR) Ion Beds, and the Airlock Heat 
Exchanger. Other potential contributors to the anomaly were the ISS environment (pH, temperatures), the EMU 
Sublimator, and EMU hardware contamination (Braycote®, system corrosion products, etc.). These investigations 
led to the ALCLR Ion Beds as potentially being the primary contributors to the high levels of silica-laded 
contaminants found in the EPS S/N 006 Water Separator drum holes. ^ 

IV. Airlock Cooling Loop Recovery Ion Bed Evaluations 
A. Airlock Cooling Loop Recovery Description 

The ALCLR water processing kit was developed as a corrective action to EMU coolant loop flow disruptions 
experienced on the ISS in May 2004 and thereafter. The components in the kit are designed to remove the 
contaminants that caused prior flow disruptions. ALCLR water processing kits have been used since 2004 as 
standard operating procedure. Periodic analysis of EMU coolant loop water and hardware examinations were used 
as a means to determine adequate functionality and optimized processing cycles as well as ALCLR component shelf 
life. 

The ALCLR water processing kit (Eigure 4) was devised to scrub and remediate the various chemical and 
biological contaminants and by-products that were found to have fouled the magnetically coupled pump in the EMU 
Transport Water Loop EPS. The heart of the kit is the EMU Ion Eilter, which is a 50:50 by volume packed bed of 
mixed anion/cation exchange resin and activated carbon. This component is periodically installed inline to the EMU 
and Airlock Heat Exchanger coolant loop and serves the purpose of removing inorganic and organic constituents 
such as nickel and iron corrosion products and organic acids with the ion exchange resin. Eurthermore, uncharged 
organic contaminants are removed with the activated carbon."^ 
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Figure 4. ALCLR processing kit components. 


In service, a 3 -micrometer filter is placed downstream of the EMU Ion Filter to capture fines from the packed 
bed prior to return of the polished water to the EMU Transport Loop. After scrubbing with the EMU Ion Filter, the 
EMU Biocide Filter is installed to add residual iodine biocide for microbial control. The EMU Biocide Filter is a 
packed bed of ion exchange resin impregnated."^ 

B. Airlock Cooling Loop Recovery Ion Bed Link to Short Extravehicular Mobility Unit 3011 Mishap 

In December 2013, five ALCLR Ion Beds and four post-Ion Bed 3-micrometer filters were returned from on- 
orbit for a complete chemical analysis. The focus of this investigation was on the proper functionality of this 
hardware since it was designed to remove the types of contaminants found in the SEMU 3011 S/N 006 EPS Water 
Separator drum holes from the EMU Transport Water Loop. 

Each of the five ALCLR Ion Beds underwent an initial free water drain, and that water underwent a complete 
chemical analysis. The ion exchange resin from each was then removed, and different aliquots underwent separate 
acid (0.5 N nitric acid) and base (0.5 N sodium hydroxide) extractions to remove the adsorbed anions and cations, 
respectively, from the resins. Separate ion exchange resin samples from each Ion Bed then underwent capacity 
testing with a variant of ASTM Method D 3 375 -95 a (Standard Test Method for Column Capacity of Particulate 
Mixed Bed Ion Exchange Materials) to determine the remaining ion exchange capacity for each resin. ^ 

The free water ('-60 mL) drained from ALCLR Ion Bed (S/N 1003) was found to have a relatively high 
dissolved silicon level (41 ppm) when compared to the others (< 1 ppm dissolved silicon), thus indicating that the 
ion exchange resin was saturated and unable to retain silicon in its common dissolved form, as ionic silicic acid. 
Furthermore, the acid/base extracts from the ALCLR Ion Bed S/N 1003 ion exchange resin was found to have 
relatively high chloride, sulfate, potassium, and silicon levels when compared to the data generated from the other 
four beds. Finally, when the ion exchange resin capacity test was done on the resin from ALCLR Ion Bed S/N 1003, 
it was found to have no remaining ion exchange capacity, and silicon immediately discharged from the ion exchange 
resin when capacity challenge sodium chloride was added (Figure 5). This was a stark difference from the other four 
Ion Beds, which all showed significant remaining ion exchange resin capacity (Figure 6).^ 
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Blue - Conductivity 
Red - Silicon Concentration 
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Figure 5. Capacity test results — Ion Bed S/N 1003 — no remaining capacity. 

Similar analyses of the entire inventory of ground and on-orbit and returned ALCLR Ion Beds determined that 
several other Ion Beds were partially or completely exhausted. This suggested the ALCLR Ion Bed S/N 1003 
contamination and partial exhaustion was not a single occurrence. An incremental root cause that linked several of 
the Ion Beds together was thereafter investigated.^ 
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Figure 6. Capacity test results — Ion Bed S/N 1013 — significant remaining capacity. 
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The free water ('-100 mL) drained from 3 -micrometer filter housing S/N 1034 was found to contain a relatively 
high dissolved silicon concentration (16 ppm) when compared to the other three 3 -micrometer filter water samples 
that were analyzed, thus indicating the Ion Bed that last interfaced with 3 -micrometer filter S/N 1034 was putting 
out relatively high silicon level. ^ 

A search of the ALCLR-related records was conducted. It was found that 3 -micrometer filter S/N 1034 was last 
used with ALCLR Ion Bed 1003 on August 14, 2012, indicating silicon was being released from ALCLR Ion Bed 
1003 at that time and then presumably flowed through 3 -micrometer filter S/N 1034 and into the SEMU EPS that 
was undergoing the ALCLR process at the time. The ALCLR-related records then showed that the ALCLR Ion Bed 
S/N 1003/3 -micrometer filter combination interfaced with SEMU 3011 on August 14, 2012 — the first indication of a 
source of silicon-rich water flowing directly into the SEMU 3011 EPS Water Separator drum. Lurthermore, ALCLR 
Ion Bed S/N 1003 interfaced two additional times with SEMU 3011 after it was determined to be exhausted on 
August 14, 2012, prior to the July 16, 2013, water-in-the-helmet mishap. What remained unanswered at that time 
was why the silicon was preferentially being released from ALCLR Ion Bed S/N 1003, why the silicon then 
precipitated in the SEMU 301 1 EPS Water Separator Drum, the reason for ALCLR Ion Bed S/N 1003 to exhibit out- 
of-family exhaustion, and the source of the out-of-family silicon levels. 

C. Silicon Retention on an Ion Exchange Resin 

A review of the literature and discussions with several subject matter experts helped provide an understanding of 
why silicon was being preferentially released from ALCLR Ion Bed S/N 1003. Silicon, as dissolved silicic acid, is a 
weakly bound anion to ion exchange resin. Other anions, such as carbonate, chloride, phosphate, and sulfate will 
displace silicic acid from anion exchange sites in a competitive situation.^ 

As previously mentioned, ALCLR Ion Bed S/N 1003 used with EMU 3011 on August 14, 2012 (S/N 1003) was 
found to be contaminated with high levels of silicon, chloride, and sulfate anions. The chloride and sulfate ions are 
expected to more strongly bind to ion exchange resin vs. ionic silicic acid, which is weakly bound It was surmised, 
at that time, that the high levels of chloride and sulfate anions would be expected to displace the normally weakly 
bound ionic silicic. The displaced ionic silica would then be expected to either move further down the scrubber bed 
to fresh anion exchange sites (if available) or would exit the scrubber bed. 

As nominal levels of more strongly bound anions continued to enter the scrubber bed, they would continue to 
displace the more weakly bound ionic silica. Nominal ionic silica eventually would run out of fresh anion exchange 
sites with which to bind as well and would exit the ion exchange bed. After the August 14 incident, ion exchange 
scrubber bed S/N 1003 was effectively an ionic and precipitated silica generator. 

D. Silicon Precipitation 

A review of the literature and discussion with several subject matter experts also helped provide an 
understanding of why silicon would precipitate in the SEMU 3011 EPS Water Separator Drum. The input suggested 
that high levels of ionic silica in water are prone to agglomeration and particle formation, particularly where water 
steams of varying pH join (the EMU Transport Water and Sublimator condensate streams merge in this area). 
Lurthermore, the input suggested that, once formed, particles of silicon would separate and migrate to the outer 
walls of the EPS Water Separator in operation due to its centrifugation action at 19,300 revolutions per minute. The 
Water Separator undergoes periodic drying during operation, which is further expected to lead to ionic silica 
exceeding solubility limits and forming silica/silicate -rich precipitates. Linally, once a layer of silica/silicate 
precipitates forms, it would be expected to attract additional silica/silicate that was introduced.^ 

Beaker level tests related to silicon solubility in water were conducted and showed that silicon solubility in 
solution was a function of pH (t Si at tpH) which is in agreement with the literature sources that were referenced 
and input from various subject matter experts. 

Shifts in water pH (start at pH 2-10, shift to pH 5 and pH 7) showed little effect on Si solubility with/without 
centrifugation indicating that potential pH shifts due to the joining of two water streams in the EMU (Sublimator 
condensate stream and Transport Water Loop) likely played a minimal if any role in the precipitation of 
silica/silicates in the SEMU 3011 Separator Drum.^^ 

The presence of metal cations (Le, Ni, Zn) at various pHs (2 - 10) showed little effect on Si solubility, 
suggesting that corrosion products from the SEMU 3011 Pump Rotor or other fluid loop sources played a minimal, 
if any, role in the precipitation of silica/silicates in the SEMU 3011 Separator. 

The presence of Braycote® showed essentially no effect on Si solubility, suggesting that the excessive Braycote® 
observed in several areas of the SEMU 3011 Transport Water Loop likely played a minimal, if any, role in 
facilitating the precipitation of silica/silicates in the SEMU 3011 Separator Drum. It should be noted, however, that 
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once precipitates formed, excessive Braycote® would be expected to act as a “trap” for precipitated silica/silicates, 
thereby serving as a potential collection point/ ^ 

Testing indicated that as water volume was reduced through evaporation, the ratio of total Si/reactive Si 
increased at pH 9 - 10, indicating colloidal Si formation, essentially the first step in the precipitation of Si. Through 
evaporation, the amount of Si-rich precipitate that formed was shown to be a function of water pH Si at tpH). So 
a basic solution with high levels of Si would result in a relatively high Si precipitate due to the water evaporation 
expected to occur in the EMU Water Separator, a feasible pathway to what occurred in the SEMU 3011 Water 
Separator Drum.^^ 

E. Contaminated Ion Bed Link to Ground Processing 

Immediately following the EVA 23 mishap, it was evident that a water cleanliness problem on orbit would 
require remediation. Since the ALCLR filters were the primary means of scrubbing the EVA water loops and the 
investigation had not yet uncovered a reason to suspect the filters, manufacture of eight additional ALCLR units 
(S/Ns 1020 - 1027) was initiated in anticipation of using them for the recovery effort. Assembly and ground 
processing of the filters was completed on December 20, 2013, and three of the filters (S/Ns 1020, 1021, and 1022) 
were shipped for flight on Orb-1 on December 23, 2013. The remaining filters were sent to Building 7 controlled 
storage.^ 

On December 26, 2013, “smudges and scratches” were noted on the exterior of the housings of the recently 
processed ALCLR filters, prompting an inspection. Eurther analysis of the hardware revealed that the housings were 
pitting and corroding, in some cases all the way through the housings (Eigure 7).^ 



Figure 7. Pitting and corrosion on the external surfaces of the ALCLR housings. 


After discovery of the corroding filters, water samples were taken from the free water in the ALCLR filters, the 
Activated Carbon/Ion Exchange (ACTEX) test stand (where the filters are processed), several DI water faucets in 
the Building 7 facility, and the output of the Building 7 deionized (DI) water processing facility. In all cases, the 
water conductivity and chloride concentrations exceeded the Type A Space Shuttle water specification, SE-S-0073, 
which was the water specification for the EMU at the time of the mishap (see Table 1 for key data outages from SE- 
S-0073).' 
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Table 1. Post ALCLR Corrosion Discrepancy Free Water Test Results 


Parameter 

ACTEX Test 
Stand 

ALCLR S/N: 
1023 

ALCLR S/N: 
1027 

LCVG Test 
Stand 

Crew 
Escape DI 
Sink 

Bldg7DI 

Facility 

Output 

SE-S-0073 

Conductivity 

(|jS/cm) 

150 

2500 

130 

107.9 

141.9 

717* 

<3.3 

Chloride (ppm) 

36 

730 

26 

23.2 

- 

- 

< 1 


These results suggested gross contamination of the ALCLR Ion Beds before they were launched to the ISS and 
identified a potential issue with the flight hardware ground test facilities. All on-orbit ALCLR activities were 
suspended at that point. ALCLR units were pulled from controlled storage and returned from the ISS for analysis, 
which confirmed that ALCLR units were transported to the ISS in a contaminated state, though they were not 
actually used on-orbit. ALCLR Ion Bed ground processing became a major focus of the mishap investigation 
thereafter. 

F. Airlock Cooling Loop Recovery Ion Bed Ground Processing 

The building 7 centralized DI water processing facility provides the water that is used to process the ALCLR Ion 
Bed. Several shortcomings of this water system were found upon further investigation, as follows:^ 

1. The water effluent conductivity sensor was set at 5.0 pS/cm vs. the requirement at that time of < 3.3 
pS/cm. A visual indicator — a simple green/red light — indicated compliance with this requirement. 

2. The system conductivity sensor was located after the last ion exchange bed (the industry standard is to 
locate the conductivity sensor before the last ion exchange bed to provide a safety buffer), allowing 
potentially poor quality water to be delivered to the processing area if no one noticed the green -to-red 
light change 

3. A single technician was assigned the task to periodically look at the conductivity indicator light in a 
remote location. There was no requirement to notify building users of this water if the red light 
indicated poor quality. Furthermore, the supplier of the system would be notified if a system ion 
exchange bed was exhausted; replacement of that bed could take several days. Finally, there was no 
backup in the event the primary technician was absent. 

4. No recheck of the quality of the DI water occurred prior to the point of interface with the ALCLR Ion 
Bed being processed, though several hundred feet of static water separated the final ion exchange bed 
from the point of hardware interface. 

Various water samples from the Building 7 ALCLR Ion Bed processing laboratory (i.e., EMU lab) including the 
test stand used for processing and several Building 7 locations were analyzed and were found to greatly exceed the 
SE-S-0073 water quality requirements for process water. Some of these samples were found to be comparable to tap 
water quality.^ 

The ion beds are processed at Johnson Space Center (JSC) in Building 7 in two phases: bed packing and bed 
flushing. After the raw material is loaded into the ion bed, the bed is flushed with 150 pounds of DI water to remove 
particulate. If there are any contaminants in the flush water, they are concentrated on the ion bed during the flush. 
The ground processing investigation found a number of configuration and process management issues with the 
Building 7 DI water system that lead to the facility DI beds passing silica and other contaminants downstream when 
the facility beds were near the end of their life. The investigation concluded that the source of the excessive silica 
and exhausted ALCLR Ion Beds was from ground processing and loading of the ALCLR ion beds.^’^ 

As long as the anion or cation resin in the ALCLR Ion Bed was properly processed with good quality water (not 
exhausted), the effluent water would be free of ions and the pH will be close to neutral in the on-orbit ALCLR 
application. Once the resin became partially or fully exhausted due to processing with poor quality water, in- 
operation ions that were previously exchanged and held to the resin would be released into the bed effluent (into the 
EMU FPS on-orbit during the ALCLR process). Since the ions would be concentrated and ordered within the bed 
length by affinity for the resin, the ions would not be released in the order they enter into the bed. Rather, they 
would be released from lower affinity to higher affinity to the resin functional group. That means all the weakly 
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ionic contaminants on the resin will be displaced first (such as silicon as silicic acid) and be released into the 
effluent due to adsorption of stronger ionic contaminants in the influent. 

V. Proximate and Root Cause Analysis 

The process used to determine the causes of the SEMU 3011 EPS failure has been vetted through numerous 
investigations of ISS hardware including the 2007 Russian computer failure, the starboard solar array rotary joint 
failure, and the external thermal control system pump module failures. A fault tree is used to determine the 
proximate cause(s), then a causal tree determines the intermediate cause(s) of the proximate cause (Eigure 8). 
Technical rationale is documented for each event and dispositioned as either a contributor or a non-contributor to the 
failure. The intermediate causes are then assessed to determine the root cause(s) by repeatedly asking why that event 
was able to manifest to failure. Definitions are consistent with NPR 8621. IB “NASA Procedural Requirements for 
Mishap and Close Call Reporting, Investigating, and Recordkeeping.”^ 



Figure 8. Relationship of fault tree to causal tree. 


The scope of the failure investigation effort was defined by the activities related to the evaluation of first a 
proximate cause fault tree, and then a causal tree. The proximate cause for this failure investigation was determined 
by a fault tree. Within days of the failure, a fault tree was created that included all credible hardware failure 
mechanisms that could result in the observed failure signature during EVA 23. The top event, or failure signature, 
was “Liquid leakage internal to EMU 3011 free volume on 07/16/2013 during EVA 23.” Potential leakage sources 
included: bodily fluids that migrated to the helmet; EMU element failures that would result in water leakage into the 
suit free volume, then migrate to the helmet; and EMU element failures that would import water directly to the 
helmet or introduce it through the vent loop. This fault tree scoped the actions necessary to determine the proximate 
cause of the failure.^ 

An action plan was created. This plan identified the investigative actions necessary to close each of the fault tree 
events. Actions included inspections, tests, analyses, documentation reviews, etc. Initially, each event on the fault tree was 
colored either yellow or orange, thus indicating that forward work was planned. Those events colored orange were given a 
higher priority, and closure rationale would be discussed in team meetings. Closure rationale was documented for each of 
the 44 basic events. The team evaluated the closure rationale to determine whether each basic event contributed to the 
failure. Events were colored green if technical information confirmed that the event was not a contributor. Events colored 
blue were determined to be non-contributors, but substantiated largely by engineering judgment. Contributors were 
colored red in the fault tree. Through evaluation of the closure rationale, the proximate cause was identified as “Water 
separator drum feed holes become plugged, redirects flow into vent circuit” (Figure 9).^ 
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Figure 9. Fault tree. 


Once the proximate cause was identified, a causal tree was developed to include all potential causes of the 
proximate cause, and all the sources of the contamination in the coolant loop that plugged the water separator drum 
feed holes, and to determine the intermediate cause(s) (Figure 10). Sources of contamination included those 
resulting when the ALCLR components are not capable of removing contaminants and become a source of silicon 
(including ion bed S/N 1003, LCVG, Water Line Vent Tube Assembly, Payload Water Reservoirs, umbilicals, or 
changes to pH); silicon in the EMU water exceeds the capability or precipitates between scrubs (from sublimator 
hydrophilic coating. Space Shuttle coolant loop, or airlock heat exchanger); the temperature profile affects the 
solubility of silicon; or static time between wet-dry cycles contributes to the accumulation of silicon. Closure 
rationale was documented for each of the 42 potential causes, and was evaluated by the team. The findings (red 
events) and significant observations (identified by rounded corners on the description box) were determined by 
dispositioning each event. ^ 



Figure 10. Causal tree. 


These findings and significant observations were then categorized by similarity of the causes as either an 
operational response, a response to water quality management, or an EMU hardware processing issue. Each finding 
and significant observation causal tree event was then mapped to these categories on a root cause determination 
matrix spreadsheet. The team then asked the “three whys” — a technique where the team asked why the event was 
allowed to manifest at least three times until an organizational factor was identified. Corrective actions were then 
identified for each root cause in a Corrective Action Plan (CAP) to ensure the failure would not be repeated. Finally, 
the high-level investigation results were documented in an event sequence diagram (Figure 11).^ 

A “list of truths” was collected throughout the investigation. Examples of these truths are the order of processing 
or use of ALCLR elements, physical barriers in the system, test results, hardware processing issues, etc. Once the 
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failure scenario, or sequence of events, was identified, it was challenged by the list of truths. The failure scenario 
was confirmed when it didn’t invalidate any of the truths.^ 

In summary, a systematic, structured approach resulted in a CAP that ensures all findings and significant 
observations are mitigated to preclude future failures. A fault tree was created to determine the proximate cause. The 
investigation was bounded by identifying the actions necessary to evaluate each event as a contributor. The failure 
mechanism was determined by dispositioning each event on the fault tree. The causal tree examined the potential 
causes of the contamination to determine the intermediate cause. All findings and significant observations were 
grouped by similarity, then the root causes were identified by asking why they were able to manifest to failure. 
Corrective actions were assigned to each root cause to preclude a repeat of this failure signature. The results were 
documented in an event sequence diagram. ^ 
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Figure 11. Event sequence diagram. 


This exercise was a valuable, systematic means to arrive at proximate and root cause of the SEMU 3011 Water 
in-the Helmet, as well as an organized means for a detailed treatment of corrective actions. 

The proximate cause was determined to be: Water separator drum feed holes became plugged, and redirected 
water flow into vent circuit. 

The primary root cause was determined to be: Poor water quality in JSC Building 7 water introduced into 
ALCLR Ion Beds during ground processing, resulting in partially and fully exhausted ALCLR Ion Beds being used 
with the on-orbit ALCLR processing of SEMU 3011. The key elements of this root cause analysis numbered 31, with 
numerous intermediated causes and significant observations. The key elements of the root cause analysis fit into 
three broad categories: Operational Responses, Water Quality Management, and EMU Hardware Processing. 
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VI. Corrective Action Plan 

The investigation concluded with a number of intermediate causes, root causes, significant observations, and 
corrective actions. A subset of these corrective actions was implemented as part of the effort to restore planned EVA 
capability. These corrective actions included: water quality and management across the EVA ground facilities; on- 
orbit recovery through water line flushing and component replacement; generation of associated safety 
documentation; properly controlling and verifying ion bed processing; and implementing on-orbit sampling and 
monitoring for the EMU/Airlock coolant loop. 

An EVA Suit Hardware Components and Processes Audit was performed in March 2014. This audit focused on 
items contained within the feed- water and coolant loop system of the EMU. These five facilities were audited: SGT; 
UTAS Windsor Locks; ILC; JSC Building 7; and JSC Building 9. Eour critical non-conformances were found and 
addressed.^ 

Eollowing the water audit, auditors developed CAPs for audit findings and closed them by providing objective 
evidence to the EVA hardware engineers. Einal CAP closures occurred at the EMU Panel. Of the 98 audit items 
assigned, 38 CAPs remain open as of mid-July 2014. All open items have closure plans and dates identified in their 
CAP response.^ 

As the investigation continued to reveal that contamination of water used in the EMU and its ancillary systems 
was a significant concern, it was determined that an update to the specifications governing water quality in the EMU 
was necessary. Historically, either the Space Shuttle Specification Fluid Procurement and Use Control Document 
(SE-S-0073) or the System Specification for the International Space Station (SSP41000) was used to control the 
quality of water in both the EMU feed- water and transport water circuits; although these where not in conflict with 
the needs of the EMU, they lacked certain parameters that were discovered as contaminants in the proximate cause 
of the SEMU 3011 water intrusion failure. As such, a new document was crafted to address water quality needs 
specific to the EMU system. JSC-66695, EMU Water Quality Specification, was written to define the water quality 
for any water used in the maintenance, processing, or testing of EMU hardware or any hardware that interfaces with 
the EMU transport water circuit and feed water circuit.^ 

The requirements found in JSC-66695 define the quality of source water prior to being used in the EMU in an 
effort to mitigate contamination failures of various components. These requirements are not intended to control 
water quality within the EMU during operations. Controlling water quality during operations is accomplished via 
maintenance, both on the ground and on-orbit. The feed-water circuit receives periodic flushing, referred to as a 
dump and fill. The transport circuit receives loop scrubs using the ALCLR hardware. Erom a technical perspective, 
the requirements found in the EMU Water Quality Specification add the need to evaluate silica, total carbon, 
bacteria count, and particulate in source water, as well as tighten the threshold of acceptable quantities of other 
contaminants when compared to the heritage Space Shuttle and Space Station documents.^ 

The EMU Water Quality Specification was written by a team comprised of chemistry and hardware experts 
throughout the NASA and contractor community. It was reviewed from April through June 2014, including multiple 
team meetings to finalize and validate the requirements. Einal signature was obtained on June 14, 2014. 

In parallel to the work that was done to create the JSC -66695 EMU Water Quality Specification, the OneEVA 
contractor began formation of a Water Management Plan to address how water is being managed throughout all the 
various facilities used during EMU processing or testing. This document was needed to respond to the changes in 
water sampling required in response of the EMU 3011 Water Intrusion Eailure. Each facility used by OneEVA and, 
in many cases, each specific test stand, maintains its own requirements and verifications to validate performance and 
water quality. The Water Management Plan describes the intent, operation, and changes required for each of these 
end items. The single greatest change throughout the processing of EVA hardware is the sampling plans, including 
ensuring sampling is consistent throughout any stage of processing or testing. The OneEVA Water Management 
Plan describes each of the facilities on contract and how the contractor will execute sampling, source water 
processing, and maintenance.^^ 

The corrective actions to this investigation continue to be worked. Key corrective actions that have been 
implemented include: the generation of an EMU-specific water quality specification and water management plan; 
the development and certification of an ALCLR Ion Bed process independent of direct Building 7 DI water; the 
implementation of numerous water quality checks prior to, during, and after the processing of ALCLR Ion Beds; the 
implementation of on-orbit sampling before and after ALCLR scrub events; the evaluation of every ALCLR Ion Bed 
and 3 -micrometer filter after flight use; and a reduction in the number of ALCLR Ion Bed uses on orbit to enhance 
safety margin. 
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VII. Summary 

The proximate cause of the U.S. EVA 23 water-in-the-helmet mishap was blockage of all water separator drum 
holes by a mixture composed primarily of silica and silicates. The blockages caused a failure of the water separator 
function that resulted in EMU cooling water spilling into the ventilation loop, migrating around the circulating fan, 
and ultimately pushing into the helmet. The root cause of the failure was determined to be JSC Building 7 ground- 
processing shortcomings of the ALCLR Ion Eilter Beds, which led to various levels of contaminants being 
introduced into the filters before they left the ground. Those contaminants were thereafter introduced on-orbit into 
the EMU hardware during ALCLR scrubbing operations. A methodical fault tree/causal tree activity was used to 
uncover 31 key elements to the root cause as well as numerous intermediate causes and significant observations. A 
regimented CAP was prepared to address all findings in a prioritized fashion to ensure a return to, and a 
sustainability of, nominal ISS EVA status. 
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